Start source compatibility suite, improve command line #226

turbolent · 2020-07-09T23:43:42Z

Start a source compatibility suite, similar to Swift's, to prevent regressions and understand what impact changes in the language and implementation have on real-world Cadence projects.

The suite contains a set of repository descriptions. When the suite is run, the repositories get cloned an checked. The runner can optionally run benchmarks and compare against another commit / branch, producing output in the terminal (pretty), as JSON, or Markdown.

In the future it we can integrate this as part of CI, maybe as a periodic job.

See the example report at the end of the PR description.
Improve the command line and REPL by providing the Flow-specific value and type declarations (e.g. getAccount), though no implementations (the implementations abort with an error message the functionality is not supported)
Add a new command line program which only checks a given program.

Example Report

Parser Errors

🙂 Still succeeding
flow-ft/contracts/ExampleToken.cdc
flow-ft/contracts/FungibleToken.cdc
flow-ft/contracts/TokenForwarding.cdc
flow-ft/transactions/burn_tokens.cdc
flow-ft/transactions/mint_tokens.cdc
flow-ft/transactions/setup_account.cdc
flow-ft/transactions/transfer_tokens.cdc
flow-nft/contracts/ExampleNFT.cdc
flow-nft/contracts/NonFungibleToken.cdc
flow-nft/transactions/mint_nft.cdc
flow-nft/transactions/setup_account.cdc
flow-nft/transactions/transfer_nft.cdc

Parser Benchmarks

😭 Regressions	OLD	NEW	DELTA	RATIO
flow-ft/transactions/burn_tokens.cdc	1081.28	1196.89	-10.69%	0.90x
flow-ft/transactions/transfer_tokens.cdc	1079.44	1130.04	-4.69%	0.96x
flow-nft/contracts/NonFungibleToken.cdc	1114.89	1161.6	-4.19%	0.96x

🎉 Improvements	OLD	NEW	DELTA	RATIO
flow-ft/contracts/ExampleToken.cdc	1182.53	1085.63	+8.19%	1.09x
flow-ft/contracts/TokenForwarding.cdc	1138.52	1019.15	+10.48%	1.12x
flow-ft/transactions/mint_tokens.cdc	1161.93	1065.76	+8.28%	1.09x

🙂 Same	OLD	NEW	DELTA	RATIO
flow-ft/contracts/FungibleToken.cdc	1095.36	1104.6	-0.84%	0.99x
flow-ft/transactions/setup_account.cdc	1144.5	1133.9	+0.93%	1.01x
flow-nft/contracts/ExampleNFT.cdc	1135.04	1137.97	-0.26%	1.00x
flow-nft/transactions/mint_nft.cdc	1198.88	1194.3	+0.38%	1.00x
flow-nft/transactions/setup_account.cdc	1208.03	1192.2	+1.31%	1.01x
flow-nft/transactions/transfer_nft.cdc	1201.75	1179.03	+1.89%	1.02x

Checker Errors

😢 Still failing	Details
flow-ft/transactions/mint_tokens.cdc	Details error: mismatched types --> /Users/bastian/Documents/work/cadence/compat/suite/flow-ft/transactions/mint_tokens.cdc:29:53

😕 Changed

Before

Now

flow-ft/contracts/TokenForwarding.cdc

Details

goroutine 1 [running]:
runtime/debug.Stack(0xc0001f9008, 0x12a08e0, 0x15ef070)
	/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/debug/stack.go:24 +0x9d
main.runPath.func1.1(0xc00016fa60, 0xc0001fbe78)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:181 +0x46
panic(0x12a08e0, 0x15ef070)
	/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/panic.go:969 +0x166
github.com/onflow/cadence/runtime/sema.(*CapabilityType).Unify(0xc00008e4d0, 0x1396080, 0xc00008e4d0, 0xc000128660, 0xc000064510, 0x968, 0x42, 0x2d, 0x970, 0x42, ...)
	/Users/bastian/Documents/work/cadence/runtime/sema/type.go:6652 +0x4c
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocationRequiredArgument(0xc000236100, 0xc00000e058, 0x1, 0x1, 0x0, 0xc00001f6c0, 0xc0000644c0, 0x1, 0x1, 0xc000128660, ...)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:481 +0x28a
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocation(0xc000236100, 0xc000718180, 0x1398860, 0xc000064390, 0xc000064390, 0x1, 0x0, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:380 +0x27f
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocationExpression.func3(...)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:97
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocationExpression(0xc000236100, 0xc000718180, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:107 +0x7af
github.com/onflow/cadence/runtime/sema.(*Checker).VisitInvocationExpression(0xc000236100, 0xc000718180, 0xc000236100, 0x1e75278)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:27 +0x3f
github.com/onflow/cadence/runtime/ast.(*InvocationExpression).AcceptExp(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:339
github.com/onflow/cadence/runtime/ast.(*InvocationExpression).Accept(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:335
github.com/onflow/cadence/runtime/sema.(*Checker).VisitCreateExpression(0xc000236100, 0xc00000c340, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_create_expression.go:37 +0xd8
github.com/onflow/cadence/runtime/ast.(*CreateExpression).AcceptExp(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:646
github.com/onflow/cadence/runtime/ast.(*CreateExpression).Accept(0xc00000c340, 0x139ba20, 0xc000236100, 0xc0001f9710, 0x1194419)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:642 +0x5f
github.com/onflow/cadence/runtime/sema.(*Checker).VisitUnaryExpression(0xc000236100, 0xc000223080, 0xc000236100, 0x1e75278)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_unary_expression.go:28 +0x62
github.com/onflow/cadence/runtime/ast.(*UnaryExpression).AcceptExp(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:507
github.com/onflow/cadence/runtime/ast.(*UnaryExpression).Accept(0xc000223080, 0x139ba20, 0xc000236100, 0xc0001f9818, 0x1009c1a)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:503 +0x62
github.com/onflow/cadence/runtime/sema.(*Checker).VisitReturnStatement(0xc000236100, 0xc00001f080, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_return_statement.go:56 +0x292
github.com/onflow/cadence/runtime/ast.(*ReturnStatement).Accept(0xc00001f080, 0x139ba20, 0xc000236100, 0x138d801, 0x138d840)
	/Users/bastian/Documents/work/cadence/runtime/ast/statement.go:36 +0x3e
github.com/onflow/cadence/runtime/sema.(*Checker).visitStatements(0xc000236100, 0xc000204200, 0x1, 0x1)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_block.go:68 +0x14b
github.com/onflow/cadence/runtime/sema.(*Checker).visitFunctionBlock.func1()
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:350 +0x4b
github.com/onflow/cadence/runtime/sema.(*Checker).visitWithPostConditions(0xc000236100, 0x0, 0x1396180, 0xc00024cc80, 0xc0001f9a90)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:308 +0x54
github.com/onflow/cadence/runtime/sema.(*Checker).visitFunctionBlock(0xc000236100, 0xc00000c380, 0xc00000cbe0, 0x1)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:343 +0xcf
github.com/onflow/cadence/runtime/sema.(*Checker).checkFunction.func2()
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:158 +0x16f
github.com/onflow/cadence/runtime/sema.(*FunctionActivations).WithFunction(0xc00026bfc0, 0xc00001f7c0, 0x3, 0xc0001f9be0)
	/Users/bastian/Documents/work/cadence/runtime/sema/function_activations.go:67 +0xfa
github.com/onflow/cadence/runtime/sema.(*Checker).checkFunction(0xc000236100, 0xc000076690, 0xc00071c900, 0xc00001f7c0, 0xc00000c380, 0x1, 0x0, 0x1)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:141 +0x17a
github.com/onflow/cadence/runtime/sema.(*Checker).visitFunctionDeclaration(0xc000236100, 0xc0007181e0, 0xc000010001, 0x12aacc0, 0xc000128510)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:77 +0x13d
github.com/onflow/cadence/runtime/sema.(*Checker).checkCompositeFunctions.func1(0xc000236100, 0xc00024cb40, 0xc0001f9d48)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:1386 +0xa2
github.com/onflow/cadence/runtime/sema.(*Checker).checkCompositeFunctions(0xc000236100, 0xc00000e060, 0x1, 0x1, 0xc00024cb40)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:1394 +0x7d
github.com/onflow/cadence/runtime/sema.(*Checker).visitCompositeDeclaration(0xc000236100, 0xc000218240, 0x2)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:115 +0x5ca
github.com/onflow/cadence/runtime/sema.(*Checker).VisitCompositeDeclaration(0xc000236100, 0xc000218240, 0x0, 0xc0000c4078)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:28 +0x3e
github.com/onflow/cadence/runtime/ast.(*CompositeDeclaration).Accept(0xc000218240, 0x139ba20, 0xc000236100, 0x2, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/ast/composite.go:41 +0x3b
github.com/onflow/cadence/runtime/sema.(*Checker).VisitProgram(0xc000236100, 0xc0000c4000, 0xc00016fc70, 0xc00016fc80)
	/Users/bastian/Documents/work/cadence/runtime/sema/checker.go:453 +0x417
github.com/onflow/cadence/runtime/ast.(*Program).Accept(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/program.go:53
github.com/onflow/cadence/runtime/sema.(*Checker).Check.func1(...)
	/Users/bastian/Documents/work/cadence/runtime/sema/checker.go:326
github.com/onflow/cadence/runtime/sema.(*Checker).Check(0xc000236100, 0x7ffeefbff8a8, 0x58)
	/Users/bastian/Documents/work/cadence/runtime/sema/checker.go:331 +0x154
main.runPath.func1(0xc00016fa60, 0xc0001fbe78, 0xc0000cca80, 0x97b, 0x7ffeefbff8a8, 0x58, 0xc0000b1118, 0xc0001fbdf0, 0xc0000b1120, 0xc0000b1108)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:190 +0x11e
main.runPath(0x7ffeefbff8a8, 0x58, 0x1, 0x7ffeefbff8a8, 0x58, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:196 +0x1a5
main.run(0xc0000b4070, 0x1, 0x1, 0x101)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:149 +0x150
main.main()
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:45 +0xbb

Details

goroutine 1 [running]:
runtime/debug.Stack(0xc0002b9008, 0x12a08e0, 0x15ef070)
	/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/debug/stack.go:24 +0x9d
main.runPath.func1.1(0xc000204ca0, 0xc0002bbe78)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:181 +0x46
panic(0x12a08e0, 0x15ef070)
	/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/panic.go:969 +0x166
github.com/onflow/cadence/runtime/sema.(*CapabilityType).Unify(0xc0000644e0, 0x1396080, 0xc0000644e0, 0xc0004a7530, 0xc000173a90, 0x968, 0x42, 0x2d, 0x970, 0x42, ...)
	/Users/bastian/Documents/work/cadence/runtime/sema/type.go:6652 +0x4c
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocationRequiredArgument(0xc000144000, 0xc0001be380, 0x1, 0x1, 0x0, 0xc000285440, 0xc000173830, 0x1, 0x1, 0xc0004a7530, ...)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:481 +0x28a
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocation(0xc000144000, 0xc0002c08a0, 0x1398860, 0xc0001b1d60, 0xc0001b1d60, 0x1, 0x0, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:380 +0x27f
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocationExpression.func3(...)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:97
github.com/onflow/cadence/runtime/sema.(*Checker).checkInvocationExpression(0xc000144000, 0xc0002c08a0, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:107 +0x7af
github.com/onflow/cadence/runtime/sema.(*Checker).VisitInvocationExpression(0xc000144000, 0xc0002c08a0, 0xc000144000, 0x1d15ea0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_invocation_expression.go:27 +0x3f
github.com/onflow/cadence/runtime/ast.(*InvocationExpression).AcceptExp(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:339
github.com/onflow/cadence/runtime/ast.(*InvocationExpression).Accept(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:335
github.com/onflow/cadence/runtime/sema.(*Checker).VisitCreateExpression(0xc000144000, 0xc0002037e0, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_create_expression.go:37 +0xd8
github.com/onflow/cadence/runtime/ast.(*CreateExpression).AcceptExp(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:646
github.com/onflow/cadence/runtime/ast.(*CreateExpression).Accept(0xc0002037e0, 0x139ba20, 0xc000144000, 0xc0002b9710, 0x1194419)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:642 +0x5f
github.com/onflow/cadence/runtime/sema.(*Checker).VisitUnaryExpression(0xc000144000, 0xc000615f50, 0xc000144000, 0x1d15ea0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_unary_expression.go:28 +0x62
github.com/onflow/cadence/runtime/ast.(*UnaryExpression).AcceptExp(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:507
github.com/onflow/cadence/runtime/ast.(*UnaryExpression).Accept(0xc000615f50, 0x139ba20, 0xc000144000, 0xc0002b9818, 0x1009c1a)
	/Users/bastian/Documents/work/cadence/runtime/ast/expression.go:503 +0x62
github.com/onflow/cadence/runtime/sema.(*Checker).VisitReturnStatement(0xc000144000, 0xc000207480, 0x0, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_return_statement.go:56 +0x292
github.com/onflow/cadence/runtime/ast.(*ReturnStatement).Accept(0xc000207480, 0x139ba20, 0xc000144000, 0x138d801, 0x138d840)
	/Users/bastian/Documents/work/cadence/runtime/ast/statement.go:36 +0x3e
github.com/onflow/cadence/runtime/sema.(*Checker).visitStatements(0xc000144000, 0xc0002050f0, 0x1, 0x1)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_block.go:68 +0x14b
github.com/onflow/cadence/runtime/sema.(*Checker).visitFunctionBlock.func1()
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:350 +0x4b
github.com/onflow/cadence/runtime/sema.(*Checker).visitWithPostConditions(0xc000144000, 0x0, 0x1396180, 0xc000212780, 0xc0002b9a90)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:308 +0x54
github.com/onflow/cadence/runtime/sema.(*Checker).visitFunctionBlock(0xc000144000, 0xc000203820, 0xc000202980, 0x1)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:343 +0xcf
github.com/onflow/cadence/runtime/sema.(*Checker).checkFunction.func2()
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:158 +0x16f
github.com/onflow/cadence/runtime/sema.(*FunctionActivations).WithFunction(0xc000295ce0, 0xc0002854c0, 0x3, 0xc0002b9be0)
	/Users/bastian/Documents/work/cadence/runtime/sema/function_activations.go:67 +0xfa
github.com/onflow/cadence/runtime/sema.(*Checker).checkFunction(0xc000144000, 0xc00020c640, 0xc000615e60, 0xc0002854c0, 0xc000203820, 0x1, 0x0, 0x1)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:141 +0x17a
github.com/onflow/cadence/runtime/sema.(*Checker).visitFunctionDeclaration(0xc000144000, 0xc0000943c0, 0xc000010001, 0x12aacc0, 0xc0004a73e0)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_function.go:77 +0x13d
github.com/onflow/cadence/runtime/sema.(*Checker).checkCompositeFunctions.func1(0xc000144000, 0xc000212640, 0xc0002b9d48)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:1386 +0xa2
github.com/onflow/cadence/runtime/sema.(*Checker).checkCompositeFunctions(0xc000144000, 0xc0001be390, 0x1, 0x1, 0xc000212640)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:1394 +0x7d
github.com/onflow/cadence/runtime/sema.(*Checker).visitCompositeDeclaration(0xc000144000, 0xc000140300, 0x2)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:115 +0x5ca
github.com/onflow/cadence/runtime/sema.(*Checker).VisitCompositeDeclaration(0xc000144000, 0xc000140300, 0x0, 0xc00070e078)
	/Users/bastian/Documents/work/cadence/runtime/sema/check_composite_declaration.go:28 +0x3e
github.com/onflow/cadence/runtime/ast.(*CompositeDeclaration).Accept(0xc000140300, 0x139ba20, 0xc000144000, 0x2, 0x0)
	/Users/bastian/Documents/work/cadence/runtime/ast/composite.go:41 +0x3b
github.com/onflow/cadence/runtime/sema.(*Checker).VisitProgram(0xc000144000, 0xc00070e000, 0xc0001c34a0, 0xc0001c34f0)
	/Users/bastian/Documents/work/cadence/runtime/sema/checker.go:453 +0x417
github.com/onflow/cadence/runtime/ast.(*Program).Accept(...)
	/Users/bastian/Documents/work/cadence/runtime/ast/program.go:53
github.com/onflow/cadence/runtime/sema.(*Checker).Check.func1(...)
	/Users/bastian/Documents/work/cadence/runtime/sema/checker.go:326
github.com/onflow/cadence/runtime/sema.(*Checker).Check(0xc000144000, 0x7ffeefbff8a8, 0x58)
	/Users/bastian/Documents/work/cadence/runtime/sema/checker.go:331 +0x154
main.runPath.func1(0xc000204ca0, 0xc0002bbe78, 0xc000122a80, 0x97b, 0x7ffeefbff8a8, 0x58, 0xc0001be268, 0xc0002bbdf0, 0xc0001be270, 0xc0001be260)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:190 +0x11e
main.runPath(0x7ffeefbff8a8, 0x58, 0x1, 0x7ffeefbff8a8, 0x58, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:196 +0x1a5
main.run(0xc00001e0b0, 0x1, 0x1, 0x101)
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:149 +0x150
main.main()
	/Users/bastian/Documents/work/cadence/runtime/cmd/check/main.go:45 +0xbb

🙂 Still succeeding
flow-ft/contracts/ExampleToken.cdc
flow-ft/contracts/FungibleToken.cdc
flow-ft/transactions/burn_tokens.cdc
flow-ft/transactions/setup_account.cdc
flow-ft/transactions/transfer_tokens.cdc
flow-nft/contracts/ExampleNFT.cdc
flow-nft/contracts/NonFungibleToken.cdc
flow-nft/transactions/mint_nft.cdc
flow-nft/transactions/setup_account.cdc
flow-nft/transactions/transfer_nft.cdc

Checker Benchmarks

🎉 Improvements	OLD	NEW	DELTA	RATIO
flow-nft/transactions/transfer_nft.cdc	1225.37	1138.09	+7.12%	1.08x

🙂 Same	OLD	NEW	DELTA	RATIO
flow-ft/contracts/ExampleToken.cdc	1219.45	1217	+0.20%	1.00x
flow-ft/contracts/FungibleToken.cdc	1131.53	1171.86	-3.56%	0.97x
flow-ft/transactions/burn_tokens.cdc	1221.45	1195.25	+2.15%	1.02x
flow-ft/transactions/setup_account.cdc	1216.85	1210.97	+0.48%	1.00x
flow-ft/transactions/transfer_tokens.cdc	1180.47	1174.43	+0.51%	1.01x
flow-nft/contracts/ExampleNFT.cdc	1180.06	1179.37	+0.06%	1.00x
flow-nft/contracts/NonFungibleToken.cdc	1134.59	1113.52	+1.86%	1.02x
flow-nft/transactions/mint_nft.cdc	1209.73	1177.89	+2.63%	1.03x
flow-nft/transactions/setup_account.cdc	1195.47	1187.57	+0.66%	1.01x

start with checking FT and NFT contracts and transactions

- add support for JSON output - make running old parser optional

- add support for JSON output - add benchmark option

benjaminkvm · 2020-07-13T18:15:51Z

runtime/cmd/cmd.go

 	i := 0
 	printErr := func(err error, filename string) {
 		if i > 0 {
 			println()
 		}
-		print(runtime.PrettyPrintError(err, filename, codes[filename], true))
+		_, writeErr := writer.Write([]byte(runtime.PrettyPrintError(err, filename, codes[filename], true)))


Appreciate the flexibility here 👍

turbolent · 2020-07-13T19:05:32Z

I've pushed more commits that extend the script. Ishould have opened a separate PR instead of adding commits to the existing PR, but I didn't realize / forgot when I commit pushed.

The PR should be ready for review now!

benjaminkvm

Nice work Bastian! I remember us talking about this before -- very cool to see! 🎉

Would be cool to see those parser stats for each opened PR actually!

turbolent · 2020-07-27T21:09:54Z

Set up a local environment with much better stable performance and documented it in setup.md.
With the documented settings, the results have much lower deltas when comparing against the same code:

Parser Benchmarks

🙂 Same	OLD	NEW	DELTA	RATIO
flow-ft/contracts/ExampleToken.cdc	1209.03	1206.01	+0.25%	1.00x
flow-ft/contracts/FungibleToken.cdc	1210.36	1217.07	-0.55%	0.99x
flow-ft/contracts/TokenForwarding.cdc	1223.11	1216.43	+0.55%	1.01x
flow-ft/transactions/burn_tokens.cdc	1196.71	1194.49	+0.19%	1.00x
flow-ft/transactions/mint_tokens.cdc	1203.24	1202.04	+0.10%	1.00x
flow-ft/transactions/setup_account.cdc	1242.56	1262.49	-1.60%	0.98x
flow-ft/transactions/transfer_tokens.cdc	1194.66	1196.53	-0.16%	1.00x
flow-nft/contracts/ExampleNFT.cdc	1206.9	1202.81	+0.34%	1.00x
flow-nft/contracts/NonFungibleToken.cdc	1210.09	1217.32	-0.60%	0.99x
flow-nft/transactions/mint_nft.cdc	1232.88	1222.74	+0.82%	1.01x
flow-nft/transactions/setup_account.cdc	1201.93	1199.8	+0.18%	1.00x
flow-nft/transactions/transfer_nft.cdc	1225.28	1208.91	+1.34%	1.01x

Checker Benchmarks

🙂 Same	OLD	NEW	DELTA	RATIO
flow-ft/contracts/ExampleToken.cdc	1113.41	1117.63	-0.38%	1.00x
flow-ft/contracts/FungibleToken.cdc	1184	1183.19	+0.07%	1.00x
flow-ft/contracts/TokenForwarding.cdc	1201.56	1199.4	+0.18%	1.00x
flow-ft/transactions/burn_tokens.cdc	1233.12	1228.88	+0.34%	1.00x
flow-ft/transactions/mint_tokens.cdc	1249.57	1246.33	+0.26%	1.00x
flow-ft/transactions/setup_account.cdc	1200.48	1203.03	-0.21%	1.00x
flow-ft/transactions/transfer_tokens.cdc	1244.52	1259.17	-1.18%	0.99x
flow-nft/contracts/ExampleNFT.cdc	1199.1	1205.55	-0.54%	0.99x
flow-nft/contracts/NonFungibleToken.cdc	1204.03	1203.7	+0.03%	1.00x
flow-nft/transactions/mint_nft.cdc	1196.99	1199.22	-0.19%	1.00x
flow-nft/transactions/setup_account.cdc	1204.86	1203.78	+0.09%	1.00x
flow-nft/transactions/transfer_nft.cdc	1190.8	1192.95	-0.18%	1.00x

turbolent added 3 commits July 9, 2020 15:54

add check command

961d1a0

import Flow builtins in REPL and command line runner

62a94de

add a source compatiblity suite

fd28c7a

start with checking FT and NFT contracts and transactions

turbolent requested review from zhangchiqing, psiemens, joshuahannan and benjaminkvm July 9, 2020 23:43

turbolent self-assigned this Jul 9, 2020

turbolent added the Improvement label Jul 9, 2020

turbolent mentioned this pull request Jul 10, 2020

Use fixed-point literal onflow/flow-ft#29

Closed

turbolent added 7 commits July 11, 2020 15:28

improve parse command

87c9db9

- add support for JSON output - make running old parser optional

improve check command

ce9f246

- add support for JSON output - add benchmark option

exit with appropriate status code

6dc655c

improve the JSON output

5a5ff44

improve the JSON output

8f68cb1

parse the JSON output from the parse and check commands

70e0a41

generate a comparison from results

1335227

benjaminkvm reviewed Jul 13, 2020

View reviewed changes

regression and improvement where swapped

9ed099e

benjaminkvm approved these changes Jul 17, 2020

View reviewed changes

turbolent added 5 commits July 27, 2020 13:45

Merge branch 'master' into bastian/source-compat-suite

43b09a2

fix imports

8affcb3

improve Python compatibility

d1f83a8

include program in result

f10a694

don't include program if run for benchmarking

1d5e61c

turbolent merged commit e9e336c into master Jul 27, 2020

turbolent deleted the bastian/source-compat-suite branch July 27, 2020 21:10

turbolent mentioned this pull request Jul 27, 2020

Document setup for compatiblity suite #270

Merged

turbolent mentioned this pull request Sep 29, 2020

Improve compat suite #362

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start source compatibility suite, improve command line #226

Start source compatibility suite, improve command line #226

turbolent commented Jul 9, 2020 •

edited

Loading

benjaminkvm Jul 13, 2020 •

edited

Loading

turbolent commented Jul 13, 2020

benjaminkvm left a comment

turbolent commented Jul 27, 2020

Start source compatibility suite, improve command line #226

Start source compatibility suite, improve command line #226

Conversation

turbolent commented Jul 9, 2020 • edited Loading

Example Report

Parser Errors

Parser Benchmarks

Checker Errors

Checker Benchmarks

benjaminkvm Jul 13, 2020 • edited Loading

Choose a reason for hiding this comment

turbolent commented Jul 13, 2020

benjaminkvm left a comment

Choose a reason for hiding this comment

turbolent commented Jul 27, 2020

Parser Benchmarks

Checker Benchmarks

turbolent commented Jul 9, 2020 •

edited

Loading

benjaminkvm Jul 13, 2020 •

edited

Loading